Overview about this project

This system manages Human genes data, three servers in JAPAN store and shares this data together. When one of three has a problem, we still can recover data from two others.

1. Manage data and all divided parts of it.
2. Allow the user to divide a file in a server into three paths and send it to the other server (by sync action).
3. Upload a file to a server
4. Download a file and merge it.

Download and upload big file (80Gb)

With the big file, if using JS as normal to upload and download, you will get out of memory. The solution is using the old school approach/origin approach (jQuery/iframe/cookie-based approach). Why? because in this way the browser will handle the memory for us. Here I learn. https://stackoverflow.com/a/9834261/7185713

Combine 3 .tar file to 1 file only

Inside the 1 .tar file, we have a hundred small files, cause the system cannot send a big file(80Gb) to others servers. They have to split it into many chunks. So when I combine 2 .tar files. The challenge I facing is:

1. I have to combine each small file or 1st .tar to each file of 2nd .tar first. Then combine all of them in order.
2. Disk space using is not more than 1 file. That means if I have two 80Gb files. The disk space I using to both save a temporary file and combine files is not larger than 80Gb.
3. Run 3-5 thread to handle combine at the same time and without conflict in shared data of thread, keep it in order, and not out of memory.

=> All about algorithm and solution